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We often assume, quite logically, a connection between professional development 1 and 
improved program quality and youth outcomes. We make a "leap of faith" (Hall & 
Gannett, 2011) that our work with practitioners will trickle down to their programs, and, 
ultimately, to the children and youth with whom they work. However, virtually no 
studies, in either the out-of-school or the school-based literature, present empirical 
evidence to support this belief. 

In the literature on out-of-school time (OST) education, the Massachusetts Afterschool 
Research Study (MARS, Miller & Hall, 2007) demonstrated a positive correlation in OST 
programs between having well-trained staff and meeting quality criteria. However, the 
study was a non-random sample of programs in one state and has not been replicated. 
The MARS study did not determine whether it was in-service professional development 


1 By professional development, I mean training that occurs once a person has been hired as a professional 
in the field. It is also called in-service training. 


Hill, Leap of Faith 


National Institute on Out-of-School Time, Wellesley, 2012 


1 


or some other type of educational experience— for example, a university degree or 
certificate program— that had a direct effect on program quality. 

A fair amount of research in the in-school literature documents the effects of 
professional development on teaching practice. Guskey (2002) outlines five levels of 
impact of professional development: participants' reactions, participants' learning, 
organization support and change, participants' use of knowledge and skill, and student 
learning outcomes. However, as Bouffard (2004) notes, "no studies have examined ... 
how professional development impacts youth" (p. 10). Guskey himself writes that 
evaluators of professional development efforts need to look for evidence, not proof, 
that professional development is effective because the "relationship between 
professional development and improvements in student learning in these real-world 
settings is far too complex and includes too many intervening variables to permit simple 
causal inferences" (p. 9). There is clearly a missing link in the chain of inference on 
impact. 

Hall and Gannett (2011) note that the literature suggests a strong correlation between 
professional development and program improvement. In evaluations of two OST 
certification programs, the authors found that the "greatest improvements were 
reported in three competency areas: activities and curriculum, child and youth 
development, and program management" (p. 18). But changes to activities and 
curriculum do not automatically translate into improved youth outcomes. 

What does the lack of evidence linking professional development causally to youth 
outcomes mean to those who fund or provide OST professional development? It seems 
there are three options: 

1. Give up on funding and providing professional development and training 
because there is no evidence that professional development has an impact 
on program quality and youth outcomes. 

2. Dedicate funding to study the effects of professional development on 
program quality and youth outcomes. Such research would be an expensive 
proposition, because it would have to study large treatment and control 
groups. 
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3. Act on faith, assuming that high-quality (as opposed to low-quality) 
professional development will have some sort of positive impact on program 
quality and youth outcomes. 2 

Currently our only choice is option 3. The next step is therefore to define the criteria for 
high-quality professional development. 


Review of Research on High-quality Professional Development 

In a review of 1,300 studies of the impact of professional development on student 
achievement (Yoon, Duncan, Lee, Scarloss, & Shapley, 2007), nine studies were 
selected, based on quality criteria, for closer scrutiny. These studies showed that 
providing teachers with more than 14 hours of professional development had a positive 
and significant effect on student achievement. The majority of professional 
development programs studied provided follow-up support to the main event, in a type 
of linked or sustained professional development. One of the major drawbacks of the 
study was variation in the types of professional development offered, so that no 
conclusions about the effect of specific kinds of professional development on youth 
outcomes could be reached (Yoon et al., 2007). 

A 2001 study (Garet, Porter, Desimone, Birman, & Yoon) intended to define high-quality 
professional development by studying outcomes of the Eisenhower Professional 
Development Program for teachers. The outcomes were self-reported increases in 
teacher knowledge and skills or changes in classroom practice. Eisenhower funds many 
different types and models of professional development, including workshops, 
conferences, study groups, professional networks and teacher collaboratives, taskforce 
work, and peer coaching. The researchers organized the data into frameworks of 
structural features— having to do with the structure or design of the professional 
development— and core features related to the substance of the training. Structural 
features include both form, such as whether the professional development is a "reform 
type" study group or a traditional workshop, and duration, which includes the total 
number of contact hours, as well as the span of time over which the activity takes 
place. 


2 Though the literature on professional development in schools is wholly occupied with academic 
outcomes, in OST we would also include nonacademic outcomes such as socio-emotional 
competencies, career or vocational preparation, and civic engagement and leadership. 
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The findings demonstrated that "time span and contact hours have a substantial 
positive influence on opportunities for active learning and coherence" (Garet et al., 
2001, p. 933). The researchers' "results indicate that sustained and intensive 
professional development is more likely to have an impact ... than is shorter 
professional development" (p. 935). Their results also indicate that professional 
development that focuses on academic subject matter (content), gives teachers active 
learning opportunities, "and is integrated into the daily life of the school (coherence), is 
more likely to produce enhanced knowledge and skills" (p. 935). The researchers found 
that professional development that encourages collective participation and professional 
communication among practitioners— in other words, that builds learning communities— 
gets better outcomes (Garet et al., 2001). 

Penuel, Fishman, Yamaguchi, and Gallagher (2007) studied 454 teachers who had 
received professional development from a variety of providers on the GLOBE 
curriculum, an international earth-science program. The data sources included surveys 
of professional development providers, teacher surveys, and implementation data from 
an online database. While it could not link the professional development with student 
outcomes, the study was able to observe fidelity of implementation of the curriculum, 
which was the researchers' independent measure. That is, researchers were able to 
document whether and how the teachers who had attended the training were 
implementing the curriculum as intended. The professional development was analyzed 
based on the categories identified by Garet and colleagues (2001), including coherence, 
active learning, and length of time, among others. The major finding was that "teacher 
perceptions of support for planning had a positive impact on teacher learning" (Penuel 
et al., 2007, p. 947). That is, when the professional developers provided time for 
teachers to discuss how to align the curriculum with their current practices and local 
standards— when the professional development was "coherent"— teachers felt more 
prepared. Another finding was that professional development that focused on content 
as well as on methods of inquiry and the scientific process was positively related to 
implementation of the curriculum. 

In a recent article, Bausman and Barry (2011) reviewed the literature on teacher 
professional learning communities (PLCs) and their effects on the college readiness of 
students. The authors argue that PLCs are "likely insufficient for meeting the new 
expectation of the Common Core Standards to increase college and workforce 
readiness" but are promising as "opportunities for teacher teams to work collaboratively 
on student learning" (p. 176). However, the researchers found that subject matter and 
how students learn content were not typically the focus of the PLCs. The authors call 
for the development of a collection of videos that offer "externally developed, research- 
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based, and standards-aligned examples of instruction" (Bausman & Barry, 2011, p. 
176). 

Research-based Criteria for High-quality Professional 
Development 

The literature suggests several criteria that define high-quality professional 
development. 

Time Span: More and Longer 

One study found that "professional development that is sustained over time and 
includes a substantial number of contact hours on a single professional development 
focus (averaging 49 hours in one multi-study review and close to 100 in another) 
results in increases in student learning" (Chung Wei, Darling-Hammond, & Adamson, 
2010, p. 3). In order to accomplish this kind of duration, professional development must 
be viewed as part of the organizational culture. The expectation is that staff will attend 
professional development, and time and funding are made available for them to do so. 
This professional development can occur in a range of contexts. For example, a week- 
long institute that provides an in-depth analysis of a teaching issue or curriculum would 
provide approximately the length and range of time that would yield results. 
Alternatively, weekly or monthly staff development over time that follows a specific 
focus— perhaps on a theme or specific content— would also satisfy this criterion. 

Coherence: Making Connections to Practice through Active Learning 
Professional development that supports educators in integrating what they learn into 
their daily practice promotes coherence. Coherence can be accomplished in a variety of 
ways and through multiple venues. It is closely aligned with active learning in which, for 
example, practitioners learn a new teaching approach or strategy, implement the new 
technique with their students during the time between professional development 
sessions, and then reflect on this strategy during the next session. Another example of 
active learning occurs when staff members analyze student work during a professional 
development session in order determine how a new strategy would help students learn. 

Focus on Content 

Though the findings are somewhat mixed with regard to what content should be the 
focus of professional development, they do show that opportunities for in-depth 
examination of a subject have more effect on youth outcomes than do more superficial 
interventions. In a review of professional development and student outcomes, Chung 
Wei and colleagues (2009) found that student achievement improved most when 
teachers were "engaged in sustained, collaborative professional development that 
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specifically focused on deepening teachers' content knowledge and instructional 
practices" (p. 5 ). 

One of the conundrums of the OST field is that staff come not only with a range of 
expertise in such diverse areas as the arts, social work, or recreation, but also with a 
range of educational levels. In addition, there are yet few undergraduate and graduate 
schools that provide a degree in OST youth work. Colleges and universities that do have 
degree-bearing youth work or youth development majors typically do not require 
students to become competent in any academic subject. The irony is that, while school 
teachers are increasingly called upon to become more proficient in subject matter, we 
expect OST staff to improve student outcomes in both academic and non-academic 
subjects without adequate subject matter training. As a field, we need to articulate and 
address this set of conflicting expectations. 

Professional Learning Community 

While professional learning communities alone cannot affect youth outcomes, they are 
cited in the literature as being one key to improving teaching practice. That is because 
communities, by definition, encourage and sustain professional dialogue and exchange 
of resources, skills, and strategies. In addition, the literature suggests a positive impact 
when teachers feel supported, which is an essential feature of well-designed 
professional learning communities. Professional learning communities also get better 
results than more limited learning opportunities because they tend to continue for 
longer periods of time. In the literature, the structures that support these communities 
are called "reform models," which can include ongoing study groups and fellowships as 
well as online social networks. 

Summary 

Though no clear link between professional development and youth outcomes has been 
established in either the in-school or the OST literature, a research-based consensus 
establishes the characteristics of high-quality professional development: It is sustained 
over a period of time, coherent, content focused, and based in a community of learners. 
These guidelines can inform OST funders and program directors as they decide how to 
invest limited resources in the OST workforce. Empirical research still needs to be 
conducted to investigate the link between high-quality professional development and 
youth outcomes. 
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Bausman, J. M., & Barry, C. (2011). Revisiting professional communities to increase 
college readiness: The importance of pedagogical content knowledge. 
Educational Researcher, 40(4), 175-178. 

This article reviews literature on teacher professional learning communities (PLCs) and 
their effects on the college readiness of students. The authors argue that, although 
PLCs are "necessary for effective schools," they are "likely insufficient for meeting the 
new expectation of the Common Core Standards to increase college and workforce 
readiness." They find that there is "mounting evidence that certain features of 
professional development also can have an impact on student achievement. These 
features include training over an extended time period ..., a focus on the subject matter 
content and how students learn that content ..., and opportunities for teacher teams to 
work collaboratively on student learning" (p. 176). However, PLCs do not typically focus 
on research on subject matter and how students learn content. The authors end by 
calling for a collection of videos that are "externally developed, research-based, and 
standards-aligned examples of instruction" (p. 176). 


Bouffard, S. (2004). Promoting quality afterschool programs through professional 
development. Harvard Family Research Project. Issues and Opportunities in Out- 
of-School Time Evaluation, Brief No. 8. 

This article makes the case for improved evaluation and research on the impact of 
professional development. Bouffard finds that "research from education and early child 
care demonstrates that professional development is related to positive social and 
cognitive outcomes for youth," but that evaluations of professional development have 
been "sparse" (p. 10). The author mentions Guskey's (2002) five-level evaluation 
framework but concurs that most evaluations collect data on the first level, participants' 


Hill, Leap of Faith 


National Institute on Out-of-School Time, Wellesley, 2012 


8 


reactions, in the form of "post-training satisfaction surveys" (p. 10). She writes, "To 
date, no studies have examined the ... level of how professional development impacts 
youth" (p. 10). 


Chung Wei, R., Darling-Hammond, L., Andree, A., Richardson, N., & Orphanos, S. 
(2009). Professional learning in the learning profession: A status report on 
teacher development in the United States and abroad. Dallas, TX: National Staff 
Development Council. 

This article is a comprehensive review of studies of the availability of professional 
development in the U.S. and abroad as well as a review of studies of what constitutes 
quality professional development. The authors reviewed both qualitative case studies 
and quantitative, random assignment control group studies. The goal of the review is to 
"examine policies and contexts that support implementation of more effective 
professional learning tied to student learning" (p. ix). The researchers group their 
findings into categories, including content of professional development, context for 
learning, design of learning experiences, and professional communities. Some of the 
relevant findings from their review include the need for sustained, in-depth professional 
development rather than "one shot workshops," opportunities for teachers to engage in 
active learning or "sense-making" activities, and "job-embedded and collaborative 
teacher learning" (p. 9). 


Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What 
makes professional development effective? Results from a national sample of 
teachers. American Educational Research Journal, 38{A ), 915-945. 

This study, intended to define high-quality professional development, states that 
"despite the size of the body of literature, relatively little systematic research has been 
conducted on the effects of professional development on improvements in teaching or 
on student outcomes" (p. 917). In addition, few studies have "explicitly compared the 
effects of different characteristics of professional development" (p. 918). The study 
collected data on the Eisenhower Professional Development Program, a federally funded 
program for teacher professional development. The Eisenhower program funds many 
different types and models of professional development, including workshops, 
conferences, study groups, professional networks and collaboratives, taskforce work, 
and peer coaching. The researchers integrated and operationalized the ideas in the 
literature on "best practices" in professional development to create a set of scales 
describing the "characteristics of activities assisted by the Eisenhower program." They 
"then empirically tested these characteristics to examine their effects on teacher 
outcomes" (p. 918). The researchers organized the data into frameworks of structural 
features— having to do with the structure or design of the professional development— 
and core features related to the substance of the training. Structural features include 
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both form, such as whether the professional development is a "reform type" study 
group or a traditional workshop, and duration, which includes the total number of 
contact hours, as well as the span of time over which the activity takes place. All of the 
teacher outcomes were self-reported increases in knowledge and skills and changes in 
classroom practice. 

The findings demonstrate that "time span and contact hours have a substantial positive 
influence on opportunities for active learning and coherence" (p.933). The findings also 
demonstrate that "activities that give greater emphasis to content and that are better 
connected to teachers' other professional development experiences and other reform 
efforts are more likely to produce enhanced knowledge and skills" (p. 933). Results also 
indicate that "professional development that focuses on academic subject matter 
(content), gives teachers opportunities for 'hands-on' work (active learning), and is 
integrated into the daily life of the school (coherence) is more likely to produce 
enhanced knowledge and skills" (p. 935). What the authors call "reform activities," such 
as study groups, get better results, because they tend to continue for longer periods of 
time. Also, the types of professional development that encourage collective participation 
and professional communication among practitioners get better outcomes. 


Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. 

Educational Leadership, 55(6), 45-51. 

This article provides a rationale for evaluating professional development: The evaluation 
can "provide meaningful information that you can use to make thoughtful, responsible 
decisions about professional development processes and effects" (p. 1). Guskey outlines 
a framework of five critical levels of professional development to evaluate: participants' 
reactions, participants' learning, organization support and change, participants' use of 
new knowledge and skills, and student learning outcomes. The framework also 
categorizes the type of data that can be collected about each of the levels. Guskey 
describes a process for planning professional development that starts with the fifth 
level, student outcomes, and designs backward to determine what professional 
development practices "will most effectively and efficiently produce those outcomes" (p. 
10). Guskey recommends that evaluators look for evidence, not proof, that professional 
development is effective because the "relationship between professional development 
and improvements in student learning in these real-world settings is far too complex 
and includes too many intervening variables to permit simple causal inferences" (p. 9). 


Hall, G., & Gannett, E. (2010). Body and soul. Reflections on two professional 
development credential pilots in Massachusetts. Afterschool Matters, 10, 13-21. 

This article reviews two pilot credential programs in Massachusetts: the School-Age 
Youth Development credential and the Professional Youth Worker credential. The article 
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begins by reviewing current literature on professional development in the youth field, 
stating that "findings from the Massachusetts Afterschool Research Study ... showed 
that, in a large sample of afterschool programs, staff development had a significant 
relationship with program quality. Programs with more highly educated and trained 
staff, both program directors and direct service workers, demonstrated highly quality 
staff engagement, youth engagement, activities and homework time" (p. 13). Research 
suggests that professional development is a requirement for program improvement. Hall 
and Gannett describe Guskey's (2000) framework of five levels of impact of professional 
development: 1) participants' reactions, 2) participants' learning, 3) organization 
support and change, 4) participants' use of knowledge and skill, and 5) student learning 
outcomes. The authors conclude that "few studies have collected youth data specifically 
tied to training or professional development. We regularly make the leap of faith that 
high-quality training and professional development have a positive impact not only on 
the professionals who receive it, but also on the youth they serve. Effects on youth may 
not be immediate; they most likely take place over time. Use of new content and 
strategies gained in training or professional development is often delayed due to 
program or personal constraints" (p. 17). In their evaluations of two certification 
programs, the authors find that the "greatest improvements were reported in three 
competency areas: activities and curriculum, child and youth development, and 
program management" (p. 18). 


Miller, S., & Hall, G. (2005). Pathways to success for youth: What counts in after- 
school. National Intercultural Center for Research in Education (INCRE) & 

National Institute on Out-of School Time (NIOST). Retrieved September 10, 

2011, from http://www.niost.org/pdf/MARSReport.pdf 

The Massachusetts Afterschool Research Study had two major goals: (1) to identify 
those program characteristics that are most closely related to high quality 
implementation and (2) to explore the links between program quality and youth 
outcomes. The study sample emphasized communities with lower-income families in 
sites funded by the 21 st CCLC program or United Way. Starting with both academic and 
non-academic outcomes, the researchers traced outcomes back to program practices. 
They aligned programs in their sample with a series of research-based quality 
indicators. One of the findings was that "programs with more highly educated staff, 
both at the program director and direct service levels, were rated significantly higher on 
program quality, including staff engagement, youth engagement, activities, and 
homework time. In addition, programs that used certified teachers and other school 
staff tended to rate higher on [the] Quality Indicator^' (p. 24) 


Palmer, K. L., Anderson, S. A., & Sabatelli, R. M. (2009). How is the afterschool field 
defining program quality? Afterschool Matters, 9, 2-12. Retrieved from 
http://www.robertbownefoundation/orq/pdf files/2009 asm fall.pdf 
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In this review of literature on program quality, the authors suggest that well-prepared 
staff with adequate resources and professional development supports can build 
afterschool programs with a safe, enriching climate; foster positive relationships with 
the youth they service; offer focused, intentional programming; and build strong 
partnerships with families, school leaders, and the larger community to enrich and 
enhance their work. 


Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. (2007). What makes 
professional development effective? Strategies that foster curriculum 
implementation. American Educational Research Journal, 44{ 4), 921-959. 

This article reports on a study of 454 teachers who received professional development 
from a variety of providers on the GLOBE curriculum, an international earth-science 
program. The data sources included surveys of professional development providers, 
teacher surveys, and implementation data from an online database. While it could not 
link the professional development with student outcomes, the study was able to 
observe fidelity of implementation of the curriculum. The professional development was 
analyzed based on commonly accepted indicators of quality practice, including 
coherence, active learning, and length of time, among others. The major finding was 
that "teacher perceptions of support for planning had a positive impact on teacher 
learning" (p. 947). That is, when the professional development provided time for 
teachers to discuss how to align the curriculum with their current practices and local 
standards, they felt more prepared. Another finding was that professional development 
that focused on content as well as inquiry and the scientific process had a positive 
relationship to implementation of the curriculum. 


Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. (2007). Reviewing 
the evidence on how teacher professional development affects student 
achievement (Issues & Answers Report, REL 2007, No. 033). Washington, DC: 
U.S. Department of Education, Institute of Education Sciences, National Center 
for Education Evaluation and Regional Assistance, Regional Educational 
Laboratory Southwest. Retrieved from http://ies.ed.QOv/ncee/edlabs 

The authors reviewed 1,300 studies of the impact of professional development on 
student achievement, and, based on criteria from the U.S. Department of Education 
What Works Clearinghouse, selected nine for closer scrutiny. One of the major 
drawbacks of the study was that there was "variability" in the professional development 
that precluded "any conclusions about the effectiveness of specific professional 
development programs" (p. 14). That is, the studies were not selected in a secondary 
screening process based on a specific professional development model. Another 
limitation is that none of the nine studies were of middle or high school teachers. 
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However, the studies of interventions consisting of more than 14 hours of professional 
development showed a positive and significant effect on student achievement. In 
addition, all but one of the interventions offered follow-up support to the main 
professional development event. 
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